Finalize Iceberg root and shim class placement [reduced-it] by gerashegalov · Pull Request #15944 · NVIDIA/cudf-spark

gerashegalov · 2026-09-10T00:22:52Z

Related to #15821.

Description

This PR completes the Iceberg class-placement milestone:

promote conditional root-safe module classes from the newest selected Spark classifier that provides each class
move five conditional Iceberg helpers into iceberg-common
retain all remaining Iceberg implementation classes in their Spark-specific directories instead of spark-shared
remove all seven remaining Iceberg entries from unshimmed-common-from-single-shim.txt
apply the same conditional-root selection to the standard and fast distribution assemblers

After this change, assembled multi-shim distribution JARs contain no Iceberg classes under spark-shared. The shared, root-safe Iceberg helpers are at the JAR root, while Spark- and Iceberg-version-specific implementations remain under sparkXYZ.

There are no user-facing configuration or API changes.

Validation:

post-upmerge clean Scala 2.12 multi-shim dist build: 334,344,353,359
post-upmerge clean Scala 2.13 multi-shim dist build: 353,359,404,413,420
package-private-access audit passed for 11 callers across all selected real Iceberg runtime worlds
final JAR inspection found 53 Iceberg class files at root, zero under spark-shared, and the remaining classes under Spark-specific directories
verified that conditional root classes have no copies under the selected shim directories
verified newest-provider precedence: general helpers came from Spark 4.1.3 and GpuBaseReader came from its newest applicable classifier, Spark 3.5.3
Spark 4.1.3 focused Iceberg unit tests: 5 passed
Spark 3.5.3/Scala 2.13 GPU integration smoke with Iceberg loaded through extraClassPath: 6 passed
fast direct multi-shim assembler completed successfully and produced the same Iceberg placement
synthetic Spark 4.1.3/3.5.3 regression passed for newest-provider selection and final dedupe placement in both assemblers
shell, Python syntax, Scala 2.13 POM synchronization, and whitespace checks passed

Checklists

Documentation

Updated for new or modified user-facing features or behaviors
No user-facing change

Testing

Added or modified tests to cover new code paths
(synthetic multi-classifier provider-selection and dedupe coverage for both assemblers)
Covered by existing tests
(IcebergDeletionVectorSuite and the Iceberg read, append, and no-op-write integration smoke tests)
Not required

Performance

Tests ran and results are added in the PR description
Issue filed with a link in the PR description
Not required

Performance testing is not required because this changes build-time source selection and JAR placement without modifying runtime execution logic.

This change and its validation were prepared with AI assistance and reviewed by a human before publication.

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com> # Conflicts: # dist/unshimmed-common-from-single-shim.txt

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com> # Conflicts: # dist/unshimmed-common-from-single-shim.txt

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

…final-layout Signed-off-by: Gera Shegalov <gshegalov@nvidia.com> # Conflicts: # dist/unshimmed-common-from-single-shim.txt

gerashegalov · 2026-09-10T15:25:19Z

build

greptile-apps · 2026-09-10T15:29:06Z

Confidence Score: 5/5

The PR appears safe to merge; the newest-provider selection is consistently implemented in both assemblers and covered by focused layout tests.

Summary

Promotes conditional root-safe classes from the newest classifier that provides each class.
Keeps Spark- and Iceberg-version-specific implementations in their shim directories.
Moves conditional Iceberg helpers into iceberg-common and synchronizes Scala 2.12/2.13 build configuration.
Adds regression coverage for both standard and fast distribution assemblers.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Process classifiers newest-first] --> B[Read root-safe module members]
  B --> C{Already promoted?}
  C -->|No| D[Promote class to JAR root]
  C -->|Yes| E[Keep newer root provider]
  D --> F[Record promoted member]
  E --> G[Process next classifier]
  F --> G
  G --> H[Binary deduplication]
  H --> I[Root-safe helpers at JAR root]
  H --> J[Version-specific Iceberg classes in sparkXYZ]

_{Reviews (2) · Last reviewed commit: "Add root-safe provider selection regress..."}

gerashegalov · 2026-09-10T15:31:10Z

build

firestarman

LGTM, just one nit.

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov · 2026-09-11T18:08:03Z

build

NvTimLiu

LGTM +1

Follow-up to #15944 and #15957. ### Description After both Iceberg packaging changes reached `main`, `package-parallel-worlds.py` began reading `system_runtime_path` from the loaded `iceberg_runtime.py` module. `RootSafeProviderSelectionTest` creates a minimal fake version of that module, but the fixture still defined only `coordinates`. As a result, the standard-assembler subtest failed with `KeyError: 'system_runtime_path'` before Maven compilation, causing install and verify matrix jobs on subsequent pull requests to fail or be canceled by fail-fast. This change adds the missing `system_runtime_path` function to the fake module. It returns `None`, matching the test fixture's existing behavior when no explicit system Iceberg runtime is configured. Production code is unchanged. Validation on current `main` (`b23ab02e986767dfd26fde2e221d58658d9a2179`): ```text $ python3 dist/scripts/tests/test_root_safe_provider_selection.py . ---------------------------------------------------------------------- Ran 1 test in 0.251s OK ``` The failure was also reproduced before the fixture change with the same `KeyError` reported by the Maven matrix in #15982. This change was prepared with AI assistance and reviewed by the author. ### Checklists Documentation - [ ] Updated for new or modified user-facing features or behaviors - [x] No user-facing change Testing - [x] Added or modified tests to cover new code paths - [ ] Covered by existing tests - [ ] Not required Performance - [ ] Tests ran and results are added in the PR description - [ ] Issue filed with a link in the PR description - [x] Not required Signed-off-by: Hongbin Ma <mahongbin@apache.org>

gerashegalov added 11 commits August 31, 2026 13:00

Move independent Iceberg helpers to root-safe module

3725ab5

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Keep Iceberg catalog wrappers runtime-specific

3ef9f1d

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Merge remote-tracking branch 'origin/main' into codex/unshim-iceberg-…

999babe

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com> # Conflicts: # dist/unshimmed-common-from-single-shim.txt

Merge remote-tracking branch 'origin/main' into codex/unshim-iceberg-…

e0a82cd

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Update row-lineage post-processor tests

230ec0b

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Merge remote-tracking branch 'origin/main' into codex/unshim-iceberg-…

d3f34b8

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com> # Conflicts: # dist/unshimmed-common-from-single-shim.txt

Merge remote-tracking branch 'origin/main' into codex/unshim-iceberg-…

ed0e8d9

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com> # Conflicts: # dist/unshimmed-common-from-single-shim.txt

Fix Iceberg reader utility test after up-merge

f131577

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Merge remote-tracking branch 'origin/main' into codex/unshim-iceberg-…

3611f1c

…independent-helpers Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Finalize Iceberg root versus shim placement

8806121

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

Update Iceberg helper copyright years

ac31e5c

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov self-assigned this Sep 10, 2026

gerashegalov mentioned this pull request Sep 10, 2026

[EPIC] Minimize version-specific production classes #15821

Open

18 tasks

Merge remote-tracking branch 'origin/main' into codex/unshim-iceberg-…

2b281bb

…final-layout Signed-off-by: Gera Shegalov <gshegalov@nvidia.com> # Conflicts: # dist/unshimmed-common-from-single-shim.txt

gerashegalov marked this pull request as ready for review September 10, 2026 15:24

gerashegalov requested a review from a team as a code owner September 10, 2026 15:24

gerashegalov added the task Work required that improves the product but is not user facing label Sep 10, 2026

gerashegalov requested a review from a team September 10, 2026 15:31

firestarman reviewed Sep 11, 2026

View reviewed changes

Comment thread dist/build/package-parallel-worlds.py

firestarman previously approved these changes Sep 11, 2026

View reviewed changes

Add root-safe provider selection regression test

31a2132

Signed-off-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov dismissed firestarman’s stale review via 31a2132 September 11, 2026 18:05

gerashegalov requested a review from firestarman September 11, 2026 18:07

NvTimLiu approved these changes Sep 14, 2026

View reviewed changes

firestarman approved these changes Sep 14, 2026

View reviewed changes

gerashegalov merged commit fad4244 into NVIDIA:main Sep 14, 2026
59 checks passed

This was referenced Sep 14, 2026

Fix root-safe provider test runtime stub [reduced-it] #15985

Merged

[BUG] Retry UCX shuffle sends after GPU OOM #15982

Open

res-life mentioned this pull request Sep 14, 2026

Fix Iceberg runtime stub in root-safe packaging test [reduced-it] #15987

Closed

8 tasks

Navigation Menu

Uh oh!

Finalize Iceberg root and shim class placement [reduced-it] - #15944

Finalize Iceberg root and shim class placement [reduced-it]#15944
gerashegalov merged 13 commits into
NVIDIA:mainfrom
gerashegalov:codex/unshim-iceberg-final-layout

gerashegalov commented Sep 10, 2026 •

edited

Loading

Uh oh!

gerashegalov commented Sep 10, 2026

Uh oh!

greptile-apps Bot commented Sep 10, 2026 •

edited

Loading

Uh oh!

gerashegalov commented Sep 10, 2026

Uh oh!

Uh oh!

firestarman left a comment

Uh oh!

gerashegalov commented Sep 11, 2026

Uh oh!

NvTimLiu left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

gerashegalov commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklists

Uh oh!

gerashegalov commented Sep 10, 2026

Uh oh!

greptile-apps Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

gerashegalov commented Sep 10, 2026

Uh oh!

Uh oh!

firestarman left a comment

Choose a reason for hiding this comment

Uh oh!

gerashegalov commented Sep 11, 2026

Uh oh!

NvTimLiu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gerashegalov commented Sep 10, 2026 •

edited

Loading

greptile-apps Bot commented Sep 10, 2026 •

edited

Loading